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© Multiprocessor cache system. 

@ The bandwidth of the data transfer among a 
main memory and snoopy caches is improved by 
solving the bus neck in a multiprocessor system 
using a snoopy cache technique. Shared bus cou- 
pling is employed for an address/ command bus 5 
requiring bus snoop whereas multiple data paths 
coupled by an interconnection network 7 are used 
for the data bus not requiring bus snoop. The mul- 
tiple data paths 7 reflect the order of the snoopy 
operations on the order of data transfer such as to 
maintain data consistency among the caches. 
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This invention relates to multiprocessor sys- 
tems and, more particularly, to multiprocessor sys- 
tems having a plurality of processors provided with 
respective private caches and having a shared 
memory space. 

Conflicts in access to a shared memory is the 
most. serious bottleneck that prevents improvement 
of the * system performance in a multiprocessor 
system of a shared memory type. In order to 
lighten the bottleneck, techniques using additional 
private caches provided for respective processors 
and thereby decreasing the required bandwidth for 
the shared memory are often used. Further a tech- 
nique for maintaining the consistency of data 
among the additional caches, or "snoopy cache" 
technique is well known. In this technique, each 
cache always monitors memory access which oc- 
curs on the shared bus (the "shared bus" herein : 
means a communication medium to which a plural- 
ity of resources are connected and which is con- 
currently shared by these resources), and performs 
appropriate operations, if necessary; to a corre- 
sponding cache block for maintenance of the con- 
sistency of data in terms of- other -caches ancJ ihe 1 ' 
main memory. Such -consistency operations "are 0 : 
implemented in hardware. This technique is ex- 
cellent because the maintenance control of data 
consistency is performed easily and at a high 
speed, and it is accordingly widely adopted. How- 
ever, the "snoopy cache" technique cannot resolve 
one major problem, i.e.; bus neck because it is 
based on a shared bus architecture, and it is ac- 
cordingly practical for only small-scaled parallel 
systems including, at maximum, ten or so proces- 
sors. • ' -~ '!"•■. 

On the other hand, as a technique for solving 
the bus neck problem, an interconnection network 
(the "interconnection network" herein means a 
communication medium to which a plurality of re- 
sources are connected and which connects them 
by one to one, or by one to some, by means of a 
switch) has been studied 1 for a long time: In a 
multiprocessor system coupled by an interconnec- 
tion network, the number- of coupling links in- 
creases with the number of processors constituting 
the system. Therefore; the interconnection network 
technology ensures a transfer bandwidth which 1 is 
proportional to • the number of processors,' and 
makes it possible to- realize a large-scaled parallel ; ' 
system including hundreds of processors. How-* 
ever, it is impossible for each private cache added 
to each processor to monitor ail memory access by 
other processors. Therefore, it is theoretically f im- ! 
possible for such a system -to perform control of 
data consistency by hardware implementing the 
"snoopy cache technique? Under these circum- 
stances, it is usual to give up consistency control 
by hardware but rely on software to perform con- 
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sistency control. In this approach, caches are con- 
trolled by software so that copies of the same 
memory address are never possessed concurrently 
by a plurality of caches. More specifically, under 
5 control of software protocol, corresponding copies 
in caches are invalidated by software instructions at 
an appropriate time to ensure that only one cache 
possesses the copy at any point- of time. 
Drawbacks of this technique are the increase in 
w load imposed by the software and the decrease in 
performance caused by static invalidation by soft- 
ware instead of dynamically optimizing the use of 
caches by hardware. 

There has been proposed a technique related 
15 to the present invention, a technique combining a 
snoopy bus and an interconnection network 
(Bhuyan, L. N.; Bao Chyn Liu; Ahrhea, I. "Analysis 
of MIN based multiprocessors with private cache 
memories," Proceedings of the 1989 international 
20 Conference on Parallel Processing, 8th to 12th 
• August, 1989, pp. 51-58): In this technique, a 
snoopy bus is provided in addition to an intercon- 
'" • •• nection network. Memory access that requires 
r communication 'among- caches for control of data 
* 25 - : consistency* is processed throuQfV'th'e' snoopy bus, 
and normal memory access 'that does' hot require 
communication among caches is processed 
- through the interconnection network, in order to 
decide whether the communication among the 
30 x caches is required, a table storing conditions of all 
shared copies in the system is added to each 
cache. In this technique, the upper limit of the 
transfer bandwidth is determined by either that of 
the shared bus used for access to shared data or' 
'* 35 that of the interconnection network used for access 
tb particular data, selected depending on which is 
saturated earlier. Therefore, the upper limit of the 
transfer bandwidth in this technique largely de- 
pends on the characteristics of a program to be 
40 executed. It is reasonable to consider that, in a , 
multiprocessor system using a snoopy cache tech- 
nique well designed so as to significantly decrease 
• the cache miss ratio, a large fraction of the whole 
access requests ' occurring on the system bus 
45 would be access requests generated by commu- 
nication among "caches for control of data consis- 
' ' tency. Therefore, this technique merely realizes a 
transfer bandwidth several' times wider than the 
bandwidth realized by the shared bus coupling 
so technique. This technique also requires that each 
cache should have a management table that de- 
scribes conditions of the entire system in order to 
make it possible to locally determine whether ac- 
* cess using the shared bus is required or only 
"55 access using the interconnection network is re- 
quired. In addition, the control mechanism of this 
technique becomes complicated because it must 
control both the shared bus and the interconnection 



2 



3NSOOCID: <£P 0610621A1J_> 



EP 0 510 821 A1 



network by using the table. 

This invention provides a multiprocessing sys- 
tem comprising a plurality , of processors; a main 
memory divided into a plurality of modules; a plu- 
rality of cache memories provided for the proces- 
sors; shared bus means . coupled to the cache 
memories for transferring data address information 
to the cache memories; control means provided for 
said cache memories for monitoring said address 
information transferred through the shared bus. 
means to perform data consistency procedures; 
and interconnection network means for ^selectively 
interconnecting the cache memories and the mem- 
ory modules,, on the basis of the address informa- 
tion, for data transfer therebetween. , 

This invention has been made in view of the 
aforementioned circumstances, and enables the. 
bus neck of the "snoopy cache technique".. based , 
on the shared bus .coupling to be removed, iby 
using simple, .hardware (cpntrol mechanism) -without 
using software on which! .multiprocessor systems r 
coupled through the interconnection t network ,woul^ 
have relied for data cppsistenqy/maintenance.-.. w 
According to th©.jjnyerifi^n K in ^a, tightly coupled n 
multiprocessor sy^ter^ r^^ir^,^.plurgHt^ .of, process 
sors provided witti ^re$p\ective, , private caqhesi % and 
having shared .rpe.mory.^ the . 

snoopy cache technique for. maintaining.-.the data, 
consistency among the caches,, the interconnection^ 
network structure can be introduced, without any 
adverse affection , to the snoopy cache : technique, 
and significant increase in the transfer bandwidth of 
the memory bus can be achieved. ... ' . ^ 

Embodiments of the invention, will now be^ de- 
scribed, by way of exarnpleipnjy, with reference to 
the accompanying drawing,, wherein:- ; 
Fig. 1 is a block diagram showing a general 
arrangement of a multiprocessor of a shared- 
bus, shared memory and snoopy cache type; 
Fig. 2 is a block diagram showing an embodi- 
ment of the invention; 

Fig. 3 is a block diagram showing an example of 

data path switch used in the embodiment; 

Fig. 4 is a timing chart of memory access and 

bus snoop of the embodiment; 

Fjg. 5 is a timing chart of memory access and 

bus snoop in a multiprocessor using , a conven- . 

tional snoopy cache technique; and 

Fig. 6 is a block diagram of, the data path switch 

for explaining the pperation of an alternative 

example. 

Fig. 1 shows a multiprocessor system, .using 
the "snoopy cache" technique. jn Fig, 1, a plurality, 
of processors 1a to in are connected, to , a shared, 
bus 3 and a shared memory 4 via. their .respective 
private caches 2a to 2n. Each of. the private caches 
2a to 2n monitors memory access occurring on the 
shared bus 3. and maintains data consistency 
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among plural caches by executing an appropriate 
operation, if necessary, to a corresponding cache 
block. That is, what is necessary in the snoopy 
cache technique are that all of the caches monitor 
address/commands on the shared bus and that the 
order of the< snoop is reflected to the order of data 
transfer sufficiently, to maintain data consistency 
among caches (this is realized without any addi- 
tional means because data is. transferred through 
the bus in. the, prde.r ; of the, snoop). Accordingly, 
monitoring the data bus itself, is not necessary. On 
the other hand, in recent, high-speed microproces- 
sors, which often use a cache, line.sizeas long as 
64 bytes or more, such a. long cache line is block- 
transferred' on the system bus having a. limited bit 
.width by using a plurality of bus cycles (for exam- 
ple, 8 bytes : x 8 cycles). That > is,: Xhe 
address/command cycle required.for the bus snoop 
is quite a short period of- 1 : , or. 2 bus: cycles; 
nevertheless, the system . bu$. is. occupied for.a : 
significantly - long , time for , transferring the • long^ 
cache line. The technique:; according to the. inven- 
tion positively -considers the above-mentioned twq, r 
facts, and^se^ i;i the ?i shaned busj coupling?. for:: the., 
address/command t bu% requiring the bus snoop,, but , 
uses multiple data, paths couple.d by- theHntercon- . 
nection network for the data bus. not requiring the 
5 bus snoop. .The multiple data paths, however, must, 
reflect the order of the snoop to the order of data 
transfer sufficiently to, maintain data consistency 
among the caches. According to this approach, 
while the snoopy cache technique is logically ap- 
plied in its entirety, the interconnection network can . 
be utilized for increasing the transfer bandwidth. « : 

An embodiment of the invention is explained 
below with reference to Figs. 2 and 3. 

In Fig. 2, snoopy caches 2a to 2n provided for 
respective processors 1 a, to 1n are coupled to- ~ 
gether by a single address/command bus 5> 
Snoopy operation for cache consistency control is., 
performed through the address/command bus : 5. 
On the other hand, , data path 6a to 6n from the 
respective snoopy, caches 2a to 2n are coupled to ■ 
a shared memory , system, which consists of a plu- 
rality of interleaved memory modules 9a to. 9m, via 
a data path switch 7 and datapaths 8a to -8m. 

Fig. 3 r .shqws an : e-xample. of . the ..data path 
switch 7 in which m y and c n, of -Fig-.. 2 are, bo,tri eight, 
-respectively,. that. is, processors la to: 1n and mem- 
ory modules 0 9a to. ,9m, a/e : 8 > sets, ^respectively. 
Multiplexers. r 10a to 10h select data paths 8a;-to 8h. 
pf the memory, modules , 9a, to 9h,\ and,, ; cojmept . 
them to the V: data paths ; . : 6a?tO;.6h-. oMhe snoopy 
caches 2a to 2h. = Multiplexers, 11a to 1 1h select the 
data paths 6a to 6h of the snoopy caches. 2a to 2h, 
and issue them , to the. datapaths -8a to 8h of the 
memory modules 9a to 9h, A data path controller 
12 controls the multiplexers 10a to 10h and 1.1 a to 
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11h on the basis of address and command on the 
address/command bus 5, and establishes a data 
path necessary for data transfer. 

Next explanation is directed to a data transfer 
method between a cache and a memory and 5 
among caches with reference to Figs. 2 and 3. 
Assume here that the cache line is 8 times wider 
than the data bus- width and the cache lines are 
interleaved into the memory modules 9a to 9h in 
such a way that cache lines are stored in order of A i'o i 
the youth of their address into the memory mod- 
ules 9a, 9b,.., 9h, 9a,.-. For example, the nth, (n + i)- 
th, and (n + 7)th cache lines are sequentially 1 " 
stored in the memory modules 9a, 9b, and 9h: 
In this case, respective parts of the address are ' 75 
used as follows: - 

A) Several least significant bits will designate " : i5 
respective bytes of data having the data' bus v 
width. They are ordinarily transferred in a- de- : ' : i: 
coded form as byte enable: r 1 ' • " :: - 20 

B) Three subsequent less significant" bits will 
designate the location of the data having ' the 
data bus width in the cache line. * • : ' t l -' • 

C) Three subsequent Mess significant brfs^will " ! - : -* ,w 
designate a' memory module' in whicK the cacHe 1 ' W 
line is located. • \ < '<■'■ M; 

D) The remaining more significant bits will des- ^ 
ignate the -location of the - cache line in the ' 
memory module;' ■ 

Therefore, predetermined several most signifi- 
' cant bits are used for changeover of the multiplex- 
ers 10a to 10h. 

1) When data are read from the memory into a 
cache: - ' 

Consider the case ' where the snoopy cache 
2d reads data from the memory module 9d. 
Using the address/command issued on the ad- ' 
dress command bus 5 by the snoopy cache 2d, 
the data path controller 12 controls the mul- 
tiplexer 10d and connects the data path 8d to ao 
the data path 6d. Through this data path (shown 
by a dotted line in the left half portion of Fig.3), 
data in the memory module 9d are read into the 
snoopy cache 2d in 8 bus cycles. 

2) When data are written from a cache into a ' 45 
memory: 

Consider the case where the snoopy cache 
2d writes data intb the ^memory ^module 9d. ' 
Using the address/command -issued on the k 
address/command issued ^oh the address com- so 
mand bus 5 by the snoopy cache 2d', the data 
path controller 12 controls the* multiplexer 11d : < ! 
and connects the data path 6d to 'the ; data patrr ' : 
8d. Through this data path (shown by a dotted ' ' 
line in the right half portion of Fig.3) from the 55 
snoopy cache '2d are written into 1 the memory - ' 
module 9d in 8 bus cycles. ' ■ 

3) Transfer of data among caches: 
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Data transfer from a cache to a cache is 
effected by writing data from a cache into an 
associated memory module and reading out 
them again. 

Fig. 4 is a timing chart which shows_ how to 
multiplex the memory access' and the bus snoop. 
The abscissa indicates the bus cycle by taking bus 
cycles 1 to 10 as an example. This example shows 
that access to a certain memory address occurred 
in the bus cycle 1, and the snoopy operation itself 
in all the caches has finished in the bus cycle i 
alone, but a long cache line is being block-trans- 
ferred by using 8 bus cycles 2 to 9. If access to a 
different memory module 'occurs in the subsequent 
bus cycle 2, the processing therefor is started, 
immediately: The snoopy operation finishes in the 
bus cycle 2 alone, and the cache line is transferred 
from the memory" system to the cache that re- 
quests it by using . 8 bus cycles 3 to 10. Hereafter, 
repetition of such conditions is shown. Therefore, 
under the practical operational' conditions, the ef- 
fective bandwidth Is determined ' by both the con- 
tention on the address/command bus and the con- 
tention on the memory module; however, under the 
idear : operatiohai^ ^9wn n m : 'Fig? 4, fhe\ 

theoretical maximum' Valuers determined by a 
snoopy cycle and a° cache line' size as easily un- 
derstood from the expression in the above timing " 
chart. For example/ 5 assuming thSt the snoopy cy- 
cle is 40ns (25 MHz), when the data bus width is 8 
bytes and the cache line size is 64 bytes, the 
upper limit of the realizable bus bandwidth is 1.6G 
bytes/second. Note tharth^ timing chart of the 
case using the conventional "snoopy cache" tech- 
nique is as shown in Fig. 5 in which the upper limit 
of the realizable bus bandwidth under the same ' 
conditions is 200 M "bytes/second. " ' ** 

Three alternative examples are described be- 
low. One of them uses various interconnection net- 
works other than a multiplexer as a data path 
switch. For example, a crossbar, omega network or 
the like may be used. However, as described be- 
fore, the multiple data paths must be such that the 
order of the snoop is reflected to the order of data 
transfer much enough to maintain data consistency 
among caches. ' * : ' - 

The second is a'techhique wfiich increases the 
velocity of data transfer from one cache to ..another. 
This is explained with reference to Figs. 2 and 3. 
Consider the case where data corresponding to the 
' memory module 9d" are transferred from the 
" snoopy cache 2a to the snoopy cache 2d. First, the 
-data path controller' 12 controls trie multiplexer i id 
and connects the data path 6a to the data path 8d. 
At the same timeV the data path controller 12 con-' 
trols the multiplexer ibd and connects the data* 
path 8d to the data path 6d (shown by a dotted line 
' in the left half portion of Fig.3). Then the data path 
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6a is connected to the data path 6d,. and data 
corresponding to the memory module 9d can be 
transferred from the snoopy cache 2a to the 
snoopy cache 2d. This modification makes it possi- 
ble to transfer data at a speed two times higher 5 
than the aforementioned .method, that is, transfer- 
ring data from one cache to another by reading 
data from the one cache to a memory module and 
afterward writing from the memory module to the 
another cache. . .... 10 

The final is a technique that changes corre- 
spondence of the cache lines with the memory - 
modules. Here again, let the length of the cache 
line be 8 times wider than the data bus. However, 
assume that 8 data piece having the bus width . 15 
constitutes a single cache line and are interleaved 
into the memory modules 9a to 9h in such a way 
that the data pieces are stored in order of the 
youth of their addresses in to the memory modules 
9a, 9b,.., and 9h. For example, data D1 , D2,.., and . . 20 
07 having the data bus width of a single cache line ,,. 
are sequentially stored in the memory modules 9a, 
9b,.., and 9h. The address at this, tjme is used as 
follows: , , -V - r , 

A) Several Jes^t^ sighi|icant, *bl*ts % vyill designate , . .,25 r 
respective bytes of a data bus width data piece. 
They are ordinarily transferred in. a decoded I, 
form as byte enable. x \\, 

B) Three subsequent less significant bits , will ■ 
designate the memory module in which the data .30 
bus width data piece is located. At the same . 
time, they will designate the location of the data, 

bus width data piece jn the' cache, line. 

C) The remaining more significant. bits. will des- . 
ignate the location . of the cache line in the . 35, 
memory module. , 

A data transfer method from the memory to a 
cache, from a cache to the memory and from one 
cache to another under this situation is explained 
with reference to Figs. 2 and 6, Although Fig. 6 is . 40 
the same as Fig! 3 except that data paths shown, 
by dotted lines are different. 

1) When data are read from memory into cache: 
Data are always read from all memory mod- 
ules. Consider the case where the snoopy cache ^45 
2d reads the address/command issued on. : the 
address/command bus 5, and the data path con- .-, 
troller 12 controls the .multipiexer i 1 0d and con- , .'. 
nects first the data path 8a to the data path 6d. 
Through this data path (the leftmost one of the. ...so, 
paths shown by dotted lines in the left 'half 
portion Fig. 6), data from the memory module 
9a are read into the snoopy . cache 2d. In the 
next bus cycje, the data path controiler 12 con-., 
trols the multiplexer 10d and. connects the, .data 55. 
path 8b to the data path 6d, Through this , data 
path (the second one from the left end of paths 
shown by dotted lines in the left half, portion of 



Fig. 6), data from the memory module 9b are 
read into the snoopy cache 2d. Similarly, data 
from the memory modules 9c to 9h are read 
into the. snoopy cache 2d. 

2) When data are written from cache into mem- 
ory: 

Data are always written into all memory 
modules. .Consider the case where the snoopy 
cache 2a reads the address/command Jssued on 
the address/command bus 5, and data path con- 
troller 12- controls the multiplexer f 1 1a and con- 
nects first the data path 6a- to the -data path 8a. 
Through this . data path (the . leftmost one of 
paths shown, with dotted lines in the right half 
portion of Fig. 6), data from .the snoopy cache 
2a are written into the memory module 9a. In 
the next bus cycle,, .the .data path controller 12 
cancels connection of the preceding cycle, con- 
trols the multiplexer . 11b, and connects the data 
path 6a to the data path 8b. Through this data 
path (the second one from the left end of paths , 
shown with dotted lines in the right half portion, 
of Fig. 6), data from the. snoopy cache 2a are< 
. written into -the memory; module 9b.. Similarly; 
data from -.the snoopy , cache 2a are written. ; into, 
the memory modules 9c to 9h. 

3) Data. transfer from one cache to another: • 

Data transfer from one cache to another is 
attained by writing data from a cache into an 
associated memory ; module and by reading 
them again. This technique permits reading data 
to be started from a cycle subsequent to the 
bus cycle in which writing data is started. . In 
addition, data transfer can also be effected by 
simultaneously establishing the writing data path 
and the reading data path as done by the. sec- 
ond alternative example. ... ..... 

The use of this technique gives such an advan-. 
tage that currently continued memory access, and 
subsequently commenced memory , access never 
conflict on the memory modules and that the aver- 
age bus bandwidth increases. However,. since this 
technique requires memory , of the same access 
time as the bus cycle, the memory system ber 
comes very expensive if existing semiconductor 
.memory is utilized. 

The bandwidth of the data . transfer among a 
main memory and. snoopy caches is improved by 
: solving the. bus . neck in a multiprocessor- system 
using a snoopy cache technique. .Shared, bus cou- 
pling is employed; fqr:an address/ command bus 4 5 
requiring bus. snoop whereas, multiple data- pat/is 
coupled by an interconnection .network 1 are used 
for the data bus not requiring ^bus.rsnopp. The 
multiple data patjis 7. reflect the. order;. of : the 
snoopy operations on the, order* ot. data transfer 
such as to maintain data consistency among the 
caches. . . ■ 
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Claims 



1. 



3. 



A multiprocessor data processing system com- 
prising a plurality of processors (1); a main 
memory divided into a plurality of modules (9); 
a plurality of cache memories (2) provided for 
the processors; shared bus means (5) coupled 
to the cache memories (2) for transferring data 
address information to the cache memories; 
control means provided for the cache memo- 
ries for monitoring the address information 
transferred through the shared bus means to 
perform data consistency procedures; and in- 
terconnection network: means (7) for selectively 
interconnecting the cache memories and mem- 
ory modules, on the basis of the address in- 
formation, for data transfer therebetween. 

A system as claimed in claim 1 wherein said- 
main memory is divided into said modules in 
accordance with addresses. 

The system as claimed in claim 1 wherein said 
memory is divided into said modules in such a 
manner that segments of a transfer data unit 
for said cache memories are distributed to said 
modules. 
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ules of said main memory on the basis of 
address information of transferred data; and 
shared bus means coupled to said cache 
memories in order for transferring said address 
information to said cache memories. 

A multiprocessor system comprising a plurality 
of processors (1); a main memory divided into 
a plurality of modules (4); a plurality of cache 
memories (2) provided for the processors (1); 
interconnection network means (7) for selec- 
tively connecting said cache memories (2) to 
said , modules on the basis of address informa- 
tion of transferred data; shared bus means (5) 
coupled to said cache memories for transfer- 
ring said address information to said cache 
memories; and, control means provided for said 
cache memories for monitoring said address 
information transferred through said shared 
bus. means to perform a desired procedure for 
consistency of stored data. 



5. 



6. 



4. A system as claimed in any preceding claim 
wherein data transfer among said cache 
memories is performed by data transfer from a 
source cache memory to said main memory 
and data transfer from said main memory to a 
destination cache memory. 

A multiprocessor system comprising a plurality 
of processors; a main memory divided into a 
plurality of modules; a plurality of cache 
memories provided for said processors; inter- 
connection network means for selectively con- 
necting said cache memories to said modules 
and to said cache memories on the basis of 
address information of transferred data; shared 
bus means coupled to said cache memories 
for transferring said address information to said 
cache memories; and control means provided 
for said cache memories for monitoring said 
address information transferred through said 
shared bus means to perform a desired proce- 
dure for consistency of stored data. 

A data transmission apparatus used in a mul- 
tiprocessor system having a plurality of pro- 
cessors, cache memories provided for said 
processors, and a main memory which is di- 
vided into a plurality of modules, comprising 
interconnection network means for selectively 
connecting said cache memories to said mod- 
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MAXIMUM BUS BANDWIDTH = BUS CLOCK x CACHE LINE SIZE 
= 25MHz x 64BYIES = 1.6G BYTES/SECOND 
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